Skip to content

[FlyDSL] fp8_mqa_logits: derive the split target from kernel occupancy - #5921

Merged
coderfeli merged 6 commits into
ROCm:mainfrom
akii96:aakif/fp8-mqa-logits-occupancy-aware-splits
Oct 25, 2026
Merged

coderfeli merged 6 commits into
ROCm:mainfrom
akii96:aakif/fp8-mqa-logits-occupancy-aware-splits

Conversation

@akii96

@akii96 akii96 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

On gfx942, fp8_mqa_logits now picks how many pieces to split each row's KV range into (grid.y) from how many blocks of the compiled kernel actually fit on a CU, instead of a fixed guess. gfx950 is unchanged.

Motivation

When there are too few query rows to fill the GPU, the kernel splits each row's KV range across extra blocks. Today the number of splits targets a fixed 4 x CU count blocks.

How many blocks fit on a CU depends on the compiled kernel, mainly its register use:

Heads Variant Blocks per CU
32 mfma_r4_w4 3
32 mfma_r4_w2 6
64 mfma_r4_w4 2
64 mfma_r4_w2 4

So one fixed target over-splits some kernels and under-splits others.

Suggested Fix

  • Read blocks per CU from the compiled kernel with FlyDSL's max_blocks_per_cu() (added in [Enh] Add CompiledFunction.max_blocks_per_cu() FlyDSL#1259, in FlyDSL 0.4.0). It is read once per kernel and cached.
  • Pick the split count with a small cost model: more splits shorten each block's work, but past one full wave of the GPU they cost another wave. block_overhead_tiles = 8 is the per-block fixed cost, fitted on MI325X.
  • While a CUDA graph is being captured, a kernel's first call cannot be queried, so it keeps the current fixed target.

No occupancy tables or direct HIP calls are added.

Test Plan

  • op_tests/test_flydsl_fp8_mqa_logits.py --scenario verify with --full, the default lane and --model-shapes.
  • op_tests/test_flydsl_fp8_mqa_logits_splits.py: the cost model and a check that FlyDSL reports occupancy before the kernel has launched.
  • Kernel time, main vs this PR, on one MI325X with FlyDSL 0.4.0:
    • HIP-graph replay with the cache flushed before each replay, median of 10 samples, best of 2 runs, main and this PR interleaved;
    • causal windows, head dim 128, the chunked-prefill shapes the DeepSeek indexer runs.

Test Results

MI325X (gfx942), FlyDSL 0.4.0, rebased on current main.

Correctness: --full 1590/1590, default 95/95, --model-shapes 54/54, splits test 6/6. Output is bit-identical to main on every shape below.

Kernel time (µs, lower is better):

Rows x KV 32 heads: main → PR Splits 64 heads: main → PR Splits
128 x 32k 87.6 → 76.3 (1.15x) 32 → 26 118.4 → 101.9 (1.16x) 32 → 19
256 x 64k 225.9 → 205.2 (1.10x) 19 → 14 289.4 → 279.2 (1.04x) 19 → 9
512 x 128k 783.7 → 652.7 (1.20x) 10 → 7 1213.4 → 1039.3 (1.17x) 10 → 9
1k x 16k 227.6 → 199.8 (1.14x) 5 → 3 336.9 → 291.6 (1.16x) 5 → 2
1k x 64k 787.9 → 660.4 (1.19x) 5 → 7 1222.1 → 1022.1 (1.20x) 5 → 7
1k x 128k 1564.3 → 1266.3 (1.24x) 5 → 7 2486.6 → 1997.0 (1.25x) 5 → 7
2k x 32k 654.6 → 606.2 (1.08x) 3 → 7 1197.5 → 984.4 (1.22x) 3 → 7
4k x 32k 1449.3 → 1188.0 (1.22x) 2 → 5 2015.4 → 2095.2 (0.96x) 2 → 1
8k x 32k 2704.1 → 2202.8 (1.23x) 1 → 6 3946.5 → 3489.0 (1.13x) 1 → 4
8k x 8k 398.4 → 359.4 (1.11x) 1 → 2 546.3 → 544.9 (1.00x) 1 → 1
16k x 16k 1345.5 → 1325.6 (1.02x) 1 → 3 2058.2 → 2016.2 (1.02x) 1 → 2

Geomean: 1.15x at 32 heads, 1.12x at 64 heads.

One shape is slower: 4k x 32k at 64 heads (0.96x). There the cost model picks 1 split where 2 is better.

@akii96
akii96 requested a review from a team September 29, 2026 01:05
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5921 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

@github-actions github-actions Bot changed the title [FlyDSL] fp8_mqa_logits: derive the split target from kernel occupancy [FlyDSL] fp8_mqa_logits: derive the split target from kernel occupancy Sep 29, 2026
@akii96
akii96 force-pushed the aakif/fp8-mqa-logits-occupancy-aware-splits branch from 43150be to 0b84266 Compare September 29, 2026 01:14
@akii96
akii96 requested a review from vpietila-amd September 29, 2026 01:20
ChuanLi1101
ChuanLi1101 previously approved these changes Oct 5, 2026

@ChuanLi1101 ChuanLi1101 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The split change looks right. On gfx942 a fixed cu_oversub cannot fit every variant: mfma_r2_w4 is 8 blocks/CU at 16 heads and 2 at 128 heads, so the same multiple over-splits one and under-splits the other. cu_count * occupancy, then picking the s that minimizes ceil(grid_x * s / effective_cus) * (ceil(W / s) + overhead), is the right replacement. gfx950 staying on the old path is also right; the writeup shows its optima are not in the region this model covers.

I checked the fallback too. The measured table is the min over build flags, and a miss falls back to occupancy 2. Both under-estimate, so the first launch under-splits instead of launching an extra wave. After the first dispatch the AMDHSA metadata takes over, and that matches HIP on all 216 gfx942 and 308 gfx950 configs. Output staying bit-identical follows from the kernel: a grid.y split is a disjoint slice of the KV window, with no reduction.

vpietila-amd
vpietila-amd previously approved these changes Oct 6, 2026

@vpietila-amd vpietila-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks good to me. I have some comments regarding the maintainability, but consider them as optional. However, please consider if the kernel occupancy calculation works correctly under HIP graph capture (compiled vs eager modes).

Comment thread aiter/ops/flydsl/kernel_occupancy.py Resolved
Comment thread aiter/ops/flydsl/kernel_occupancy.py Resolved
Comment thread aiter/ops/flydsl/kernel_occupancy.py Resolved
Comment thread aiter/ops/flydsl/kernel_occupancy.py Resolved
@zufayu
zufayu requested a review from yadaish October 7, 2026 01:48
@akii96
akii96 dismissed stale reviews from vpietila-amd and ChuanLi1101 via a764709 October 7, 2026 13:44
@akii96
akii96 force-pushed the aakif/fp8-mqa-logits-occupancy-aware-splits branch 2 times, most recently from a764709 to abec67b Compare October 7, 2026 13:44
ChuanLi1101
ChuanLi1101 previously approved these changes Oct 7, 2026

@ChuanLi1101 ChuanLi1101 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new push only answers the review comments. The split formula is unchanged, gfx950 stays on the old path, and CI is green on MI300X and MI355X. Approving again.

@valarLip
valarLip requested a review from coderfeli October 8, 2026 08:36
Comment thread aiter/ops/flydsl/kernel_occupancy.py Resolved
@akii96

akii96 commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @coderfeli for the help and discussion, addressed both comments:

Please don't merge yet: this needs FlyDSL 0.4.0, and aiter still pins 0.3.4.1, so gfx942 fp8_mqa_logits CI will be red until the pin is bumped. I'll update the PR once 0.4.0 is on PyPI.

_auto_num_splits targets a fixed multiple of the CU count, but the number of
blocks a CU actually holds is a property of the compiled kernel, not of the
device. On gfx942 the mfma_r2_w4 variant is 8 blocks/CU at num_heads=16 and 2
at num_heads=128, so one cu_oversub constant is correct for one kernel instance
and wrong for the rest: at 16x CU it over-splits the low-occupancy kernels (19
splits where 7 is optimal) and under-splits the high-occupancy ones (3 where 8
is optimal).

Replace the constant with the effective CU count, cu_count * occupancy, and
pick the split that minimizes a wave-quantized cost model:

    cost(s) = ceil(grid_x * s / effective_cus) * (ceil(W / s) + overhead)

Occupancy comes from the kernel itself. FlyDSL JITs, so there is no
hipFuncAttributes handle before the artifact exists, but the artifact carries
the AMDHSA metadata that determines occupancy; kernel_occupancy reads it and
applies the hardware allocation rules. That result equals
hipModuleOccupancyMaxActiveBlocksPerMultiprocessor on all 216 buildable gfx942
configurations and all 308 on gfx950. A measured table covers the first launch
of a shape, which necessarily asks before any artifact exists, and a
conservative default covers anything else; both fall on the under-estimate
side, which under-splits rather than oversubscribes.

The two arches allocate AGPRs differently and the model is parameterized for
it: CDNA3 shares one register file between VGPRs and AGPRs, CDNA4 splits them.
Summing them on gfx950 mispredicts the bkv256 variants (vgpr=452/agpr=200),
which the sum puts over the 512-VGPR file at occupancy 1 where the hardware
reports 2. Every gfx942 kernel in this op reports agpr=0, so the distinction
changes nothing there.

gfx950 keeps the fixed-oversubscription path. Its occupancy is verified, but
its own measurements do not justify switching: its auto-selected variants span
only two occupancy values against six on gfx942, its optima sit 8-24x past one
effective wave (load imbalance across causal windows, which this model does not
represent), and 32 of 35 shapes have four or more split choices within 3% of
optimal against a ~2% noise floor. Rationale recorded in _SPLIT_POLICIES.

Perf, gfx942/MI325X, over the shapes vLLM issues for chunked long-context
indexer prefill (seq_len pinned near 1024 by VLLM_SPARSE_INDEXER_MAX_LOGITS_MB,
seq_len_kv sweeping 16k-128k, causal), on top of ROCm#4963:

  DeepSeek-V4.1-Flash (index_n_heads=64): 1.25x, and 1.14x over a flat 16x
  GLM-5.3-Flash       (index_n_heads=32): 1.19x, and 1.07x over a flat 16x

ROCm#4963 is roughly neutral on DSv4.1 across that family, so nearly all of its
1.24x combined gain comes from this change; on GLM the two compose to 1.74x.
On aiter's own --model-shapes grid (block-diagonal batch_causal) the same
change is 1.08x over main with no shape regressing beyond its measured spread.

block_overhead_tiles is fitted over 86 shapes spanning three regimes -- vLLM
long-context prefill, block-diagonal batch_causal, and wide-window causal --
scored with the regimes weighted equally rather than by sample count, since
counts reflect how much was measured rather than what matters. Fitted on wide
causal alone it lands at 3, which then costs up to 1.21x on the narrow windows
production actually issues, because per-block fixed cost is a far larger share
of the work at 32 tiles than at 1024. The joint optimum is 8 under equal,
vllm-weighted and production-only weightings alike, and the worst case falls
from 1.21x to 1.07x.

Output is bit-identical to both prior policies at every in-window position on
every shape; splitting a row's window across grid.y is pure parallelism.
aiter's own sweep passes 1590/1590 cases.

Adds op_tests/flydsl/gen_fp8_mqa_logits_occupancy.py, which regenerates the
table and re-checks the model against HIP, and 25 unit tests for the selection
logic, the occupancy fallback chain, and the per-arch register-file rules.

Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Trim comments, cache the occupancy on the launcher and the split choice,
read the public CompiledArtifact.ir, load HIP through load_hip_runtime,
and add a test that reads the compiled kernel's metadata against the
measured occupancy table.

Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Replace the measured occupancy table, the IR metadata parser and the HIP generator script with CompiledFunction.max_blocks_per_cu() (ROCm/FlyDSL#1259), compiled with meta tensors once per launcher. Falls back to the cu_oversub rule on a first call inside graph capture.

Requires FlyDSL >= 0.4.0. Do not merge before aiter's flydsl pin is bumped to it.

Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Name the split search cap (_MAX_SPLITS), drop stale numbers from the _auto_num_splits docstring, and have the occupancy test call _max_blocks_per_cu directly instead of monkeypatching _auto_num_splits.

Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
@akii96
akii96 force-pushed the aakif/fp8-mqa-logits-occupancy-aware-splits branch from fbe14d0 to 0a8998f Compare October 15, 2026 01:04
@zufayu
zufayu requested a review from coderfeli October 15, 2026 06:35
@zufayu zufayu assigned coderfeli and unassigned yadaish Oct 15, 2026
@zufayu
zufayu requested a review from zhiding512 October 15, 2026 07:54
@akii96

akii96 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Quick status check on this PR as of 22nd Oct

on the FlyDSL dependency: flydsl==0.4.0 has been pinned on main since #6424 (merged 2026-10-14), so about 8 days and has remained pinned

re-checking the work on MI325X with FlyDSL 0.4.0: --full 1590/1590, default 95/95, --model-shapes 54/54 ran today

on CI: rebased against main and all checks pass

on reviewer feedback: addressed all open items

I think this is ready to merge if you're happy with how it looks now @coderfeli 🙏

@coderfeli
coderfeli merged commit 8d047ce into ROCm:main Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants